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Abstract Text - ABTX (1) : 

A system for and a method of providing an expandable home 
automation controller is disclosed which supports multiple numbers 
and multiple different types of data communications with both 
appliances and subsystems within the home as well as systems external 
to the home. The system is based upon a central processor, such as a 
microprocessor-based computer, and is connected by means of a data 
bus to control the various products and subsystems within a home or 
commercial building, such as lighting systems, security systems, 
various sensors, multiple external terminals, as well as to allow for 
the input of commands by a variety of means such as touchscreens , 
voice recogniti o n systems, telephones, custom switches or any device 
capable of providing an input to a computer system. The system 
functions can be readily controlled by the user utilizing a high 
resolution graphics display and associated touchscreen interface. 

Brief Summary Text - BSTX (7) : 

With the innovative expansion capabilities of the inventive 
system, simultaneous operation of multiple types of user devices can 
now be achieved. For example, the home automation system described 
herein may be connected to simple keyboards, serial data keypads, 
touchscreens, voice recognition circuitry, hand-held remote controls, 
computer keyboards or telephones. In fact, virtually any type of 
electronic subsystem may be connected, by means of an appropriate 
interface, to the present system. 

Brief Summary Text - BSTX (17) : 

A further object of the present invention is to provide an 
expandable home automation control system in which voice recogm i 
is utilized by the user to instruct the system to control subsystems 
present in the home . 

Brief Summary Text - BSTX (18) : 

A further object of the present invention is to provide an 
expandable home automation system in which vo i ce recogn i. '•- \ -n may be 
used in concert with high resolution color graphics displays in order 
to provide the user with an easy to use interface for instructing r.hp 
system to perform its control or monitoring functions. 
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Brief Summary Text - BSTX (25) : 

A further object of the present invention is to provide an 
expandable home automation control system which utilizes an 
electronic interface to a multi-zone security system to thereby allow 
touch or cursor-control of the security sysr.em by means of graphics 
displays . 

Brief Summary Text - BSTX (28) : 

The external custom interfaces are connected directly to various 
external systems, while the external network interfaces are connected 
directly to multiple external systems. By means of such structure, an 
80286-based AT compatible microcomputer can be utilized, by means of 
the IBM AT data bus, to control various products and subsystems 
within a home or commercial building, such as lighting systems, 
security systems, various sensors, multiple external terminals, and 
to allow for input of commands by a variety of means such as 
touchscreens , voice recognition, telephones, custom switches or any 
device capable of providing input to a computer system. 

Brief Summary Text - BSTX (29) : 

The method of the present invention is carried out by the use of a 
Home Environment Automated Real-time Transactor (HEART) which is a 
standardized, modular, software program that is configured for each 
installation. Secondary processors are utilized, under HEART control , 
to relay information to the central 80286-based microcomputer or 
central processor, or to translate central processor commands into 
commands their dedicated devices can understand. A star hardware 
topology is currently utilized. The secondary processors manage the 
voice recogn it i o n and voice synthesis subsystems, telephone 
communication subsystems, touchscreen subsystems, hand -held remote 
control unit communications, input/output control and monitoring, 
security/fire safety and monitoring and other intelligent subsystems 
such as lighting, audio, video and HVAC . 

Drawing Description Text - DRTX (3) : 

FIG. 2 is an schematic block diagram of the security system 
interface used as part of the apparatus of FIG. 1 ; 

Drawing Description Text - DRTX (22): 

FIG. 5c is a flow chart showing the voice recognition function of 
the Polling Loop module of the present invention. 

Detailed Description Text - DETX (9) : 
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The security sy s tem 38 used in the automated environment, may 
preferably be a Silent Knight Model 4000 security system, available 
from Silent Knight , of Minneapolis, Minn. Such a home securi; y <-••/ = •■ 
38 is connected to an RS-232 -to -security s ystem translator 36 which 
is in turn connected to the central processor 10 by means of the 
serial interface 14 and a data bus 12. It can provide simultaneous 
RS-232 communications with eight separate external systems, when thus 
configured, the serial interface 14 can communicate directly with any 
external system, such as an Elographics touchscreen controller, or 
any other system that uses standard RS-232 protocols. If, as is the 
case in the preferred embodiment, the external device uses a 
nonstandard protocol, then an RS-232 protocol converter, such as 
element 36, can be used to convert between the RS-232 protocol and 
the protocol utilized by the device to be controlled. That same 
interface, as is shown as element 14 in FIG. 1, may also be used to 
connect the central processor 10 and its data bus 12 through a 
protocol converter 34, if necessary, to various home automation buses 
such as CEBUS and SMART HOUSE. 

Detailed Description Text - DETX (14) : 

Voice reco gnition circuitry may also be connected by means of the 
bus 12 to the central processor 10. A speech processor 58, such as, 
the Model TISPEECH available from Texas Instruments, Dallas, Tex., 
may be connected to one or more remote microphones 64, such as the 
Crown PZM microphone, available from Crown International, Inc., of 
Elkhart, Ind. Such remote microphones provide a means by which the 
user of the present expandable home automation system can communicate 
with the system by requesting tasks by voice. The home automation 
system may provide voice communications to the user by means of one 
or more remote speaker 66, which in turn are connected to the speech 
processor 58 . 

Detailed Description Text - DETX (30) : 

VOICE RECOGNITION CIRCUITRY 

Detailed Description Text - DETX (31) : 

The use of voice recognition is an ideal user interface device for 
a home automation system because it allows the user to control 
functions in the home simply by speaking. Voice recognition is also 
useful in certain commercial environments. The use of the high 
resolution graphics display together with voice recognition allows 
the present home automation system to provide high quality visual 
cues to the user of the present control options, feedback of whether 
the user's command phrase has been recognized and the results 
produced for the user as the result of a commanded function. 
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Detailed Description Text - DETX (32) : 

One of the difficulties with the use of voice recognition in home 
and other environments is that such systems must perform under at 
least low level background noise and must be able to extract key 
words or phrases out of a speech or general noise background. The 
inventive expandable home automation system meets these performance 
specifications by the use of certain hardware, together with software 
to enhance the performance of the voice recognition functions in the 
home environment, as well a high resolution graphics display system. 

Detailed Description Text - DETX (33) : 

The implementation of the voice recognition system described 
herein may be accomplished by means of the central processor 10 and 
its interconnection along bus 12 to a speech processor 58 which in 
turn is connected to a remote microphone 64 and a remote speaker 66 . 
The central processor 10 may be, for example, an AT compatible 80286 
based microcomputer, Model CAT-902 from Diversified Technology, of 
Jackson, Miss, and a Model CHI 626B AT- compat ible passive backplane 
system bus, also available from Diversified Technology. As previously- 
discussed, a hard disk storage device is connected to the 
microcomputer for permanent storage of trained recognition 
vocabularies. In addition, an extended RAM memory, as previously 
described, is also provided for connection to the microcomputer to 
provide rapid loading of new vocabulary data. 

Detailed Description Text - DETX (34) : 

The speech processor 58 may preferably be embodied in the present 
system by one or more Texas Instruments TISPEECH speech processing 
boards for IBM and compatible personal computers. The system software 
is available from Texas Instruments as Model TISPEECH, which performs 
the basic v oi ce recogni t ; on , vocabulary loading, and microphone 
control functions. 

Detailed Description Text - DETX (36) : 

In order to effectively utilize the voice recognition system of 
the present invention, a graphics interface 54, which may be a Mode] 
VIP VGA video graphics display generator, available from ATI 
Technologies, is connected directly to the central processor 10 by- 
means of the bus 12. The graphics display interface 54 is used to 
drive a video display monitor 56, which may preferably a Mitsubishi 
AUM-1381A high resolution color graphics display monitor. A separate 
video graphics display generator and high resolution color graphics 
display monitor are provided at each voic e recognition area in the 
home, together with a separate microphone and speaker to effectuate 
control and aural feedback. Also, as described later herein, a 
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portion of the software which operates the central processor 10 is 
dedicated to optimizing the voice recognition parameters of the 
speech processor 58 in order to maximize performance in a home 
automation or other environment. 

Detailed Description Text - DETX (37) : 

Using such a system, the user's voice can be utilized in the 
present home automation system to provide either immediate or 
scheduled control of complete living moods, which are comprised of 
macros of multiple commands, audio/video entertainment system 
control, such as selection of the desired audio or video source, 
selection of the source parameters, such as channel selection, play, 
fast forward, rewind, volume, base, treble, balance control, etc., as 
well as the audio distribution network, for example, selecting which 
set of speakers the audio or video system will send its output to, as 
well as providing a muting function for each of those sets of 
speakers. In addition, the system can also both control and switch 
between a plurality of security video cameras, including panning, 
tilting, zooming of such security cameras. Finally, the voice 
recognition functions of the present home automation system can be 
utilized to control a complex home security system 38 having over 32 
zones, which is connected to the central processor 10. 

Detailed Description Text - DETX (38) : 

When the voice recognition functions of the present home 
automation system are utilized in concert with the high resolution 
color graphics display 56, immediate or scheduled control of many 
home automation features can be accomplished. Such features include 
direct control of the lighting within the home, or, if preferred, 
controlling the lighting mood (i.e. choosing from one of a 
predetermined number of preset lighting levels) either in one room or 
throughout many rooms of a home. Such a system can select complete 
living moods, which consists of macros of multiple commands, for 
various lighting, temperature, open or closed draperies, skylight 
settings, and entertainment system settings that have been 
predetermined and stored as data files. 

Detailed Description Text - DETX (39) : 

The voice r e cognition system, in connection with the high 
resolution color graphic display 56, can also provide home appliance 
control as well as security system control for components connected 
to the central processor 10, as has been previously described. In 
addition, the audio/video entertainment system controls and audio 
distribution network described above in connection with the use of 
only a voice recogn iti on system can obviously be controlled with the 
both the high resolution color graphics display and voice recognition 
systems . 
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Detailed Description Text - DETX (40) : 

Additional functions which can be controlled with the combination 
of voice recognition and high resolution color graphics displays are 
the security camera switching and control, security camera function 
and the complex security system 38, as previously described above. 
Also, local and remote information retrieval, such as time and 
weather and access to various remote data bases can be achieved using 
the voic e r ecognition and high resolution color graphics display 
combination. Control of the locks for various doors of the home as 
well as bath and spa controls, for example, to run the bath, set the 
temperature and turn on any pumps or motors, as previously described, 
can also be achieved by the use of the voice recognition system and 
high resolution color graphics display system combination. In 
addition, the telephone system can also be controlled by that 
combination . 

Detailed Description Text - DETX (42) : 

The present home automation system, as has been previously 
described, by means of its extended bus and ability to interface with 
a wide variety of devices, can support any user device that can be 
controlled serially, controlled with a parallel port, or with a 
custom interface compatible with the standard AT bus used by the 
inventive home automation system. Since both the bus and number of 
serial or parallel ports can be increased as desired, the number of 
interface devices and thus the number of devices in the home 
environment or commercial environment that can be controlled can also 
be increased as desired. That allows the disclosed home automation 
system to be configured to meet the exact needs of each home owner o:: 
business user. The system can thus be tailored to the layout of the 
environment to be controlled. For example, a touchscreen display can 
be provided in the kitchen, a touch switch with nine buttons can be 
placed near the exterior doors, voice recogn : r. ion can be installed in 
the master bedroom and any inside telephone can be used to control 
the system. 

Detailed Description Text - DETX (43) : 

It should also be remembered that an important aspect of the 
present invention is the commonality of use between input devices. 
For example, both the computer keyboard 18 and the voice recogn i i. ;.c; 
system use the touchscreen displays for cues and visual feedback . 
Wall-mounted telephone touch-tone buttons can be used to control a 
subset of the touchscreen commands. In addition, although not shown 
specifically in FIG. 1, other user interfaces may be added to the 
home automation system, such as track balls, mice, joy sticks, light 
pens and other cursor control devices. In addition, while a hand- he] d 
remote control unit is presently used, wireless versions of any of 
the present interfaces shown in FIG. 1, such as wireless portable 
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touchscreens and wireless microphones or radios for remote voice 
reco gnition, may also be used. Voic_e recognition may also be used 
over the telephone when that technology becomes available. 

Detailed Description Text - DETX (44) : 

In order to implement the use of multiple types of user devices in 
a modular manner which allow for different types of devices in 
different parts of the home to be used to control the home automation 
system described herein, the following equipment may be 
simultaneously connected to the home automation system central 
processor: (1) a standard PC computer keyboard 18; (2) plurality of 
dry contact switches 28a and 28b may be connected to the central 
processor 10 by means of a solid state input module and a pull up 
resistor to a Metrabyte MSSR-32 solid state relay board which is 
shown as process controller 26b, which is in turn connected to the 
parallel interface 24b, which may be an MDB-64 parallel controller 
connected to the AT bus 12 of the central processor 10; (3) multiple 
Elographics Accutouch touchscreens 16a and 16b connected serially to 
the central processor 10 by means of the Digicom COM/8 multiport 
serial interface 14; (4) a plurality of hand-held remote receivers 20 
may be serially connected to the central processor 10 by means of the 
serial interface 14; (5) a telephone system 62 with a plurality of 
hand sets, or a single line standard telephone system may be 
connected to the central processor 10 through a telephone interface 
60 and the speech processor 58 and the AT bus 12; (6) multiple voicr 
recognition locations may be connected to the central processor 10 
through Crown PZM microphones or other remote microphone systems 64 
which are in turn connected to respective TISPEECH speech processors 
58 running standard voic e recognition software which themselves are 
connected by the AT bus 12 to the central processor 10; and (7) 
multiple voice response locations with remote speakers 66 which 
provide spoken information and instructional cues to the user. All of 
the foregoing equipment is simultaneously connected to the central 
processor 10 and may be used to perform a plurality of functions. 

Detailed Description Text - DETX (48) : 

Functions that can be scheduled in this manner include turning a 
light or appliance on or off, watering the lawn, turning on different 
lighting moods at different times of the day, turning on audio music: 
or a television station, VCR recording of television programs, 
operating modes or features of the home automation system itself, 
operating modes of a security system, electric locks, reminders for 
the system user, operating modes of a telephone system and automatic 
information retrieval. As will be described further herein, the same 
types of displays may be used directly to control entertainment 
systems by working with a graphics representation of the 
entertainment system and its distributed speakers and sources . The 
user controls the system by selecting his audio or video source from 
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a graphics display and then selecting the rooms in which he would 
like the audio and video played. 

Detailed Description Text - DETX (59) : 

From the touchscreen shown in FIG. 3g, a security event log 
function may also be selected, the screen of which is shown in FIG. 
31. Also, the air conditioning or heating blowers may be reenabled by 
the user after they have been automatically shut down by the 
automation system as the result of a smoke or fire detection by the 
security system . 

Detailed Description Text - DETX (60) : 

FIGS. 3h and 3i show the sub-menu touchscreens which appear when 
the entry level function is selected from the security management 
sub-menu touchscreen FIG. 3g . Referring first to FIG. 3h, it can be 
seen that the screen is dominated by a floor plan schematic of the 
entry level section of the home. By touching the appropriate secured 
zone, the user can arm the security system, or disable a particular 
zone, depending upon the current state of the security system and the 
selected zone. As shown in FIG. 3h, the user has touched the zone 14 
which is the right living room window. The status information at the 
bottom of the screen, under the zone and name of the zone 
information, indicates that the zone is currently enabled. The system 
status information to the left of the floor layout indicates that the 
system is ready to arm. In a block above the system status 
information, a zone key is provided. 

Detailed Description Text - DETX (61) : 

The sub-menu touchscreen shown in FIG. 3i is similar to the 
touchscreen shown in FIG. 3h, with the exception that the status of 
zone 14 is now disabled. Thus, the function box provided on the right 
hand portion of the screen is one which would allow that zone to be 
enabled, instead of allowing the zone to be disabled, as shown in 
FIG. 3h. In the upper left hand corner of the floor layout of FIG. 
3i, it can be seen that the user has also disabled another window. 
Thus, the screen graphically shows the present status of all security- 
zones on the system. In order to arm the security system, the user 
touches the "ARM SYSTEM" box on FIG. 3i and the touchscreen of FIG. 
3j appears. 

Detailed Description Text - DETX (63) : 

FIG. 3k shows a sub-menu touchscreen similar to that shown in 
FIGS. 3i and 3h, except that the status of the security system 'is now 
armed. Thus, in the lower left portion of the screen, the status 
"armed" is shown in place of the "disarmed" status shown in FIGS. 3i 
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and 3h. 

Detailed Description Text - DETX (71) : 

As previously described, this entire sequence can be accompanied 
by verbal prompts, generated by the voice recognition and speech 
synthesis circuitry 58, to prompt the user through the scheduling 
scheme. Many different types of events can be scheduled in this 
manner, such as a lighting system event, control of the audio/video 
entertainment system, the energy management system, individual 
appliances and the configuration of the system itself. Thus, system 
features and functions can also be scheduled by the user, such as 
spoken alerts, pass codes, etc. 

Detailed Description Text - DETX (81) : 

The foregoing types of information can also be requested with any- 
wired or wireless cursor control or general input device, such as a 
mouse, touchscreen 16, light pen, joy stick, keyboard 18 or voice 
recognition system. 

Detailed Description Text - DETX (87) : 

INTERFACE TO A MULTI-ZONE SECURITY SYSTEM 

Detailed Description Text - DETX (88) : 

Another capability of the present home automation system is its 
ability to interface to multi-zone security systems and to allow user 
control of such security systems by means of interactive touchscreens 
16 and other alternative control interfaces. The present home 
automation system is designed to interface to and take advantage of 
the advanced features offered by sophisticated security systems Thus, 
the home automation system is designed to operate with an external 
security system, which provides several advantages . 

Detailed Description Text - DETX (89) : 

First, the present system can be performing control tasks while 
the security system is calling the alarm service. One of those 
control tasks can be contacting someone else by telephone as a backup 
to the alarm service. Another advantage of being designed to operate 
with an external se curity system is that the present home automation 
system can monitor the performance of the security system and can 
keep a separate log of all security events. Still another advantage 
is in the use and power of the interaction by the user with the 
securit y system by means of a high resolution color graphics display 
and cursor control devices. Such an interaction allows the user 
interface to be greatly improved and simplified over those previously 
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known. Further, the home automation system, since it is designed to 
operate with external security systems, can be utilized with the very- 
latest security systems and can take advantage of improvements and 
updates to such security systems. 

Detailed Description Text - DETX (90) : 

Another advantage of integrating a security system into the 
present home automation system is the ability to provide information 
to the user while announcing the alarm. The system can speak the 
location of the intruder or fire while displaying that location on a 
floor plan screen of the home. It also turns on all lights in the 
house to light exit routes and shut down heating system blowers to 
minimize smoke circulation in the house. 

Detailed Description Text - DETX (91) : 

A preferred external security s ystem to which the present home 
automation system may be interfaced is the Silent Knight 4 000 
security system, available from Silent Knight. That home securi ry 
system is shown as element 38 in FIG. 1. It is interfaced to the 
central processor 10 by means of a custom RS -232 - to- securi ty system 
translator 36, which in turn is connected to the serial interface 34a 
and then through the AT bus 12 to the central processor 10. The 
interface or translator 36 is a microprocessor-based translator and 
controller . 

Detailed Description Text - DETX (92) : 

In addition to connecting the home automation system to the 
security s ys tem bus, it listens to and interprets all data 
instructions generated by the security system on its bus, and 
translates commands from the home automation system to the security 
systems' , data protocol so that the secu ri ty system will receive the 
commands and respond appropriately. Besides allowing the security 
system 38 to receive and respond to commands from the central 
processor 10, which are ultimately generated by the user of the 
system, the home automation system is also able to record security 
events on a real time basis. 

Detailed Description Text - DETX (93) : 

As shown in FIG. 2, a four wire bus is utilized by the Silent 
Knight home se c urit y system 38, two wires for power and the other two 
for data transmission. The interface translator 36 connects to all 
four wires of that bus. Data originating in the security sysrem 38 is 
first level shifted from a data stream between +12 volts and ground 
to a data stream between +12 volts and -12 volts and then passed to 
an input bus of a Z80-based microcomputer. Under control of an EPROM 
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resident program, the Z80 microprocessor translates data and 
instructions appearing on the bus. Such data and instructions ex.it 
the translator 36 in RS-232 format, from which they ultimately reach 
the central processor 10. When the home automation system gives a 
command to the security system, it passes through the Z8 0 
microprocessor, is converted to the proper format and timing for the 
security bus, passes through another level shifter and is then 
converted to the +12 and ground format used by the security system 
38 . 

Detailed Description Text - DETX (98) : 

As has been described herein, the present expandable home 
automation system is a distributed processor system, in which a 
central processor 10 is integrated with a collection of secondary 
processors associated with the various interface elements. The 
disclosed home automation system is preferably arranged with a star 
topology, in which the secondary processors serve to relay 
information to the central processor, or to translate commands 
received from the central processor into commands that their 
dedicated devices can understand. As has been described, the 
secondary processors manage the following subsystems: voice 
recognition and voice synthesis; telephone communication; touchscreen 
communication; hand-held remote control unit communication; input and 
output control and monitoring; security and fire system safety and 
monitoring; and other optional intelligent subsystems, such as 
lighting, audio and video and HVAC . 

Detailed Description Text - DETX (102) : 

A flow chart of the functions performed by the Subsystem Interrupt 
Handler is shown in FIG. 4b. All of the tasks and devices based on a 
serial line, such as the touchscreen, hand-held remote control unit 
and security system,/ are supported by the interrupt handler, running 
in the background. When a byte of data is received at any serial 
port, an interrupt request line is raised, which causes the interrupt 
handler to execute. The interrupt handler determines which serial 
port caused the interrupt, copies the new data into that port's queue 
and then increments that queue's pointer. The Polling Loop, which is 
shown in FIGS. 5a and 5b, monitors the queue pointer and jumps to the 
queue ' s task when the queue pointer moves . 

Detailed Description Text - DETX (116) : 

6 . Voice recognition board 0 
Detailed Description Text - DETX (117) : 

7- Voice reco gniti on board 1 (optional) 
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Detailed Description Text - DETX (131) : 

FIG. 5c is a flow chart showing the voice recognition portion of 
the Polling Loop. After it has been determined that a key of the 
keyboard 524 has not been pressed, a determination is made as to 
whether the voice recognition circuitry is enabled 554. If that 
determination is affirmative, then a determination is made as to 
whether a telephone call is currently active 556. If a telephone call 
is currently active, then the system skips to the touchscreen 1 queue 
at step 558. 

Detailed Description Text - DETX (134) : 

If the current state is determined to not be the neutral state, 
then the system checks voice r ecognition to determine whether a 
template number has been returned from the currently active voire 
recognition board 580 at step 578. After steps 570 and 574, if it is 
determined that a template number has been returned, the program 
proceeds to again determine whether a template number has been 
returned at 578. If a template number has been returned, then the 
recognition score is displayed 582 and the simulated touch 
coordinates are set to the middle of the screen 584 . The system then 
jumps to the master touchscreen immediate response module 586. 

Detailed Description Text - DETX (13 5) : 

In the event that no template number has been returned at step 578 
or if the voice recog nit i o n is determined to not have been enabled, 
the system moves to the touchscreen 1 queue 558. 

Detailed Description Text - DETX (142) : 

Task 4: Master touchscreen and voice recognition boards 

Detailed Description Text - DETX (175) : 

If, as discussed in connection with FIG. 7c, the main menu state 
is initialized at 786, the master touchscreen subroutine jumps to the 
initialization entry point 786 shown on FIG. 7d, which is a flow 
chart of the general Contextual State of the master touchscreen task 
or subroutine. After the initialization entry point, the system sends 
a "prompt" phrase over the speaker 788, such as "enter your 
selection", or any other phrase preselected by the user and stored as 
a data file on the system hard drive. The general Contextual State 
master touchscreen subroutine then loads the main menu screen and 
indicates the status of the menu options, if necessary 790. The 
speech reco gn ition vocabulary, if applicable, is then loaded 792 and 
the system then sets the state to the current Contextual State entry- 
point 794 and returns to the top of the Polling Loop 500. 
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Detailed Description Text - DETX (184) : 

FIGS. 7g through 7n depict a flow chart for the security floor 
plan sub-menus for the master touchscreen subroutine. At the 
initialization entry point 410, the user has selected a security/fire 
option from the main menu and has selected from the security 
management menu shown in FIG. 3g one of the three levels shown of the 
home. The security floor plan sub-menu subroutine then stores the 
selected floor number, loads the selected floor plan screen data, 
speaks a responsive phrase through the speaker and voice synl: hes : z .1 ; , • 
system, such as "Please wait for response form the security sysi.err" 
and then displays a "please wait" message 412. The subroutine then 
sends a "zone map request" to the security/fire interface shown and 
described in connection with FIGS. 8a-8c, sets the Secur 1 ty/ Fi re Task 
to the "zone map request" state; and inserts a "security response 
expected" ISIS event into the ISIS queue 414. The Master Touchscreen 
Task state is then set 416 and the system returns to the top of the 
Polling Loop 500. 

Detailed Description Text - DETX (187) : 

Depending upon the result determined at step 426, the system may 
go directly to decode the armed or ready conditions of the seem :i • y 
system and determine that the s ecurity s ystem is armed or ready at 
step 428. Or, it may reach step 428 either directly or after 
informing the user of the response received at step 426. For example, 
if no response was received at step 426, the system speaks the phrase 
"no response" at 430 and then moves to step 428. If, on the other 
hand, an "unexpected response" was received at step 426, then the 
system will inform the user by speaking the phrase "unexpected 
response" 432, before moving onto step 428. 

Detailed Description Text - DETX (189) : 

FIG. 7h shows the entry points for the redisplay security floor 
plans function 301, the show zones, status 303 and the show secur i: 
system status 305, in addition to continuing the flow chart from the 
substate decoder 438. When the redisplay security floor plans 301 is 
jumped to, the system sets the fade color to black, loads the floor 
plan screen according to the current floor number and then sets the 
fade color to normal 307. The show zones, status 303 entry point 
occurs after step 307. The program then decodes the zones, status 309 
and marks the zones on the screen as well as displaying the status of 
any open or shunted zones 311. After step 311, the system reaches the 
show securit y system status point 305. The security floor plan sub- 
menu routine of the master touchscreen task then shows the armed or 
ready status on the screen of the monitor 313 and determines whether 
the user has previously selected a zone 315. If the user had selected 
a zone at step 315, the selected zone is marked 317 and the print 
zone status subroutine is called at 319. If no zone is selected at 
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step 315, the phrase "no zone selected" is printed or displayed on 
the screen 321. 

Detailed Description Text - DETX (197) : 

If the arm/disarm system function box is selected at step 4 52, the 
security floor plan sub-menu subroutine branches to the arm/disarm 
system substate or subroutine 458 and first removes the current 
"screen time out" ISIS event and sets the touch flag to 0 at 470. A 
determination is then made as to whether the security system is 
currently armed 472. If the security system is currently armed, then 
the arm flag is reset at 474. If the security system is not current] y 
armed, then the arm flag is set at 476. After the arm flag is either 
set or reset at steps 476 or 474, the voice synthesis system causes 
the speaker to transmit the phrase "Please enter your security pass 
code" to prompt the user to input the appropriate security sysrem 
password so that the system will permit a change from the armed or 
disarmed state to the other. The user either speaks or in some other 
manner inputs the password and the subroutine then calls a Get 
Password subroutine 480 which captures the password. I f no password 
is entered (a "null" password) 482, then the current security floor 
plan is redisplayed at 466. 

Detailed Description Text - DETX (19 8) : 

If a password is captured by the home automation system, then a 
determination is made as to whether the arm flag is set 484. If the 
arm flag is set, then an arm security system message and the password 
are sent to the Security/Fire Interface. The Security/Fire Task state 
is then set to the "arm system" state 488 (step 899 in FIG. 8c) . If 
it is determined that the arm flag is not set at step 484, then a 
disarm security system message and the captured password are sent to 
the Security/Fire Interface 490 and the Security/Fire Task state is 
then set to the "disarm system" state 492 (step 870 in FIG. 8c) . 
After the setting of the Security/Fire Task state to either the "arm 
system" or "disarm system" states at 488 or 492, respectively, the 
subroutine sets the "Security Response Expected" ISIS event at 4 94 
and informs the user to please wait by speaking that phrase 4 96 as 
well as displaying that message on the monitor 4 98. 

Detailed Description Text - DETX (204) : 

If the shunt zone substate is selected at 462, as shown in FIG. 
7i, then the program jumps to the shunt zone subroutine shown at 
point 459 in FIG. 71. The shunt zone subroutine then removes the 
"screen time out" ISIS event and says that sets the touch flag to 0 
457. A determination is then made as to whether the seem i '. y sy.m • 
is currently armed at 455. If the determination of that decision is 
affirmative, then the system speaks the phrase "Please enter your 
security pass code" 453 so that the user may enter the appropriate 
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password. The get password subroutine is called in order to capture 
the password entered by the user and then a determination is made as 
to whether a null password has been entered 451. If a null password 
has been entered at point 451, then the appropriate security floor 
plan is redisplayed 466. If the password entered is a potential 
password, then the shunt zone command and the password are sent to 
the security/fire interface 477 and the "security response expected" 
ISIS event and the zone number for the ISIS event are stored 44 9. 

Detailed Description Text - DETX (205) : 

If, at step 455, a determination is made that the security sys: e::. 
is unarmed, a shunt zone message is sent to the security/fire 
interface 447 and the ISIS event is set to "security response 
expected" and the zone number for the ISIS event is stored 44 9. After 
storing the zone number for the ISIS event, whether the security 
system is armed or unarmed, the shunt zone subroutine then proceeds 
to set the Security/Fire Task to the "shunt zone" state 44 5 and to 
inform the user to wait by both speaking the phrase "please wait for 
response form the security system " and by displaying "Please wait" as 
a message on the monitor screen 443. The appropriate Master 
Touchscreen Task substate is then set 441 to prevent further user 
actions until the security communication is resolved, and the program 
then returns to the Polling Loop 500. 

Detailed Description Text - DETX (210) : 

The system then determines whether the security system is ready at 
417. If the secur ity system is ready, then a determination is made as 
to whether the sec urity system was ready before the shunting occurred 
at 415. If the determination is made that the system was not ready 
before the shunting occurred, then the phrase "now ready" is spoken 
and the security floor plans subroutine continues to determine 
whether the securi ty system is armed 411. 

Detailed Description Text - DETX (211) : 

If a determination is made at point 417 that the security system 
is not ready, then a determination is made as to whether the security 
system was not ready before the shunting occurred 409. If the outcome 
of that determination is negative, then the phrase "now not ready" is 
spoken and the security floor plans subroutine then moves to a 
determination of whether the security sys tem is armed at 411. In the 
event that the secur i t y s ystem was not ready before shunting at 4 09 
or whether the se c urity s ystem was ready before shunting at 4 15, the 
security floor plans subroutine then moves to determine whether the 
security system is armed at 411. 

Detailed Description Text - DETX (212) : 
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If the security system is armed at 411, then a determination is 
made as to whether the security system was already armed before the 
shunting 405. If that determination is negative, then the phrase "now 
disarmed" is spoken 403 and the system then determines whether the 
security system is armed 401. 

Detailed Description Text - DETX (213): 

If the determination at step 411 is that the security system is 
not armed, then a determination is made as to whether the secur i t y 
system was disarmed before the shunting occurred at 300. If the 
outcome of that determination is negative, the phrase "now disarmed" 
is spoken 302 and a determination is again made at 401 as to whether 
the security sy s t e m is armed. If the outcome of the determination of 
whether the security system was disarmed before shunting at 300 is 
affirmative, the security floor plan subroutine then determines 
whether the security system is armed at 411. If the secuncy system 
is armed at 411, then the current security floor plan is redisplayed 
304 . 

Detailed Description Text - DETX (214) : 

If the security system is not armed at point 401, then the "please 
wait" message displayed on the screen is erased 306, the zone marker 
is erased and the current highlighted box on the screen is blanked at 
308 and the system then displays the security system status at 310. 

Detailed Description Text - DETX (225) : 

If the state at step 860 is to determined to be the "disarm 
system" state, then the program determines what type of command was 
received at step 870 and branches to the appropriate point. If the 
command received in the disarm system state is an acknowledgment, 
then the program sets the result equal to good at step 872 and then 
jumps to the Master Touchscreen Task disarm security system substate 
880. (Step 491 at FIG. 7k) If the command at step 870 determines that 
the system has already been disarmed, then the result is set to 
already disarmed at step 874 and the program then jumps to the master 
touchscreen disarm system substate at step 880. If the command at 
step 870 is determined to be the rejection of the password inputted 
by the user, then the result is set to equal password rejected at 
step 876 and the program then moves to step 880. (Step 4 91 at FIG. 
7k) If the command at step 870 is not one of the three commands 
already discussed, then the program goes to the Neutral State at step 
878 . 

Detailed Description Text - DETX (228) : 

Lastly, if the Contextual State determination at step 860 
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indicates that the current state is the arm system state, then a 
determination is made at step 899 as to which command has been 
received. If an acknowledge command has been received at step 899, 
then the result is set to good at step 897 and the program then goes 
to the master touchscreen task arm/disarm security system substate 
895. If the command received at step 899 is the already armed 
command, then the result is set to equal already armed at step 893 
and the program then goes to the Master Touchscreen Task arm/disarm 
security system substate 895. If the command at step 899 is 
determined to be not ready, then the result is set equal to not ready 
at step 891 and the program then proceeds to step 895. If the command 
at step 899 is determined to be the password rejected command, then 
the result is set equal to rejected password (a rejected password 
means the same as "bad password" in FIG. 7k) and the program goes co 
the Master Touchscreen Task arm/disarm security system substate 895. 
(Step 491 in FIG. 7k) If the command at step 899 is determined to be 
any other command than those previously discussed, then the program 
executes to the Neutral State 878. 

Detailed Description Text - DETX (234) : 

If at step 1314 it is determined that the detected security- 
response expected ISIS event is not being executed for the first 
time, then result is set to equal "no response" at 1320 and, 
depending upon the current master touchscreen state determination at 
step 1322, the subroutine jumps to the zone map request entry point 
(step 424 in FIG. 7g) at 1324, the arm/disarm security system 
substate entry point (step 491 in FIG. 7k) at 1326 or the shunt zone 
substate entry point (step 439 in FIG. 7m) at 1328. 

Detailed Description Text - DETX (23 5) : 

FIG. 13c shows the flow chart for the Wait For Bad Password 
subroutine of the ISIS events. When a "wait for bad password" ISIS 
event is detected at 1330, the subroutine sets the result to "no bad 
password message" during delay 1332 and then sets the Security/Fire 
Task state to Neutral 1334. A determination is then made of the 
current master touchscreen state 1336. Depending upon the current 
state, the system either then moves to the arm/disarm securi cy sys- «= ~ 
substate entry point 1326 or to the shunt zone substate entry point: 
1328. 

Detailed Description Text - DETX (239) : 

The following discussion describes the Arm/Disarm Security System 
algorithm, assuming no errors occur. 

Detailed Description Text - DETX (241) : 
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The Security/Fire Task sets the result to "good" 872 and then 
jumps to the arm/disarm s e curity system entry point 872 in FIG. 8b or 
point 897 in FIG. 8c. 

Detailed Description Text - DETX (243) : 

In the event that the password sent was an invalid password, then 
the Security/Fire Interface will send a "bad password" message, as 
shown on FIG. 8b at step 876 and FIG. 8c at step 889. If the 
Security/Fire Task receives a bad password message, then it will set 
the result equal to "bad password" and jump to the arm/disarm 
security system entry point at 876 in FIG. 8b or point 889 in FIG. 
8c. If the result is a "bad password", then the system is caused to 
speak the phrase "S ec urity system reports an invalid password. Please 
try again", at step 475 in FIG. 7k. 

Detailed Description Text - DETX (244) : 

Once the "wait for bad password" ISIS event times out at step 1330 
in FIG. 13c, the result will be set to "no bad password message 
during delay" at step 1330, the Security/Fire Interface state wi] 1 be 
set to Neutral at step 1334 and the system will then jump to the 
arm/disarm security system entry point 1326. 

Detailed Description Text - DETX (245) : 

Since the result is "no bad password message during delay", a zone 
map request is sent to the Security/Fire Interface at step 483 in 
FIG. 7k and the substate is set to prevent touches from initiating 
any new action, as described earlier in connection with FIG. 7k, step 
483. Those steps are performed in order to determine the current 
armed or ready state of the security system. 

Detailed Description Text - DETX (246) : 

The Security/Fire Interface then sends the requested zone. The 
Security/Fire Task sets the result equal to "zone map returned" and 
jumps to the arm/disarm secu r i ty system entry point at FIG. 8c, steps 
894 and 898. The zone map is returned prior to step 467 in FIG. 7k. 

Detailed Description Text - DETX (247) : 

Depending upon the various combinations of current and previous 
armed and ready conditions of the s_ecurity system, the system 
continues, and, by performing steps 467, 465, 463 and 461, informs 
the user of any changes . 

Detailed Description Text - DETX (251) : 
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Since the result is set to "good", a "wait for bad password" ISIS 
event is inserted at step 487 in FIG. 7m and then the master 
touchscreen substate is set to show a cursor but not compare user 
touches with the display boxes, at step 433. The purpose of the "wait 
for bad password" ISIS event is to allow the Security/Fire Interface 
time to respond with a "bad password" message in the event that the 
password sent is invalid, or the security system was armed and no 
password was sent with the shunt zone command. 

Detailed Description Text - DETX (253) : 

If the result is "bad password", then the system speaks "Sec ; :: ;• 
system reports an invalid password. Please try again" at step 429 in 
FIG. 7m. 

Detailed Description Text - DETX (292) : 

In order to describe the operation of the disclosed home 
automation system in normal use, the following describes how the user- 
would utilize the touchscreen to control the security system, it 
should be understood, however, that controlling the security system 
is only one of the functions as described herein, but is believed to 
be representative of how the instant home automation system operates. 
The following example shows, how, using touchscreen commands, a user 
would disable a security zone, and then arm his security system. 

Detailed Description Text - DETX (298) : 

In this example, the user will disable a single security zone so 
that a window can be left open, and then arm the security system 
using a floor plan screen display. To do this, the user touches the 
menu box labelled ENTRY LEVEL on FIG. 3g, since the zone to be 
disabled is on that floor. The menu box is highlighted, and when the 
user lifts his finger, the display changes to show a floor plan of 
the entry level of the home, FIG. 3h. 

Detailed Description Text - DETX (300) : 

In order to disable the right living room window, the user first 
selects that window by touching it on the floor plan display. The 
system responds by highlighting the window with a colored box, and 
describing the associated security zone and its status at the bottom 
of the screen. To disable the zone, the user then touches the white 
box labelled disable zone. (See FIG. 7i) . The system responds by 
requesting the home security sy stem to disable the zone and awaits 
confirmation of its request from the security system. (See FIG. 7.1) . 

Detailed Description Text - DETX (302) : 
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The next step in this example is for the user to arm the seci.i.'.; : y 
system . To do that, the user touches the white box labelled ARM 
SYSTEM. (See FIG. 7i) . The box is highlighted, and the display 
changes to a pass code display as shown in FIG. 3n. (The flow chart 
of FIG. 7j calls another Contextual State, which is shown in FIG. 
7d) . 

Detailed Description Text - DETX (303) : 

The next step is for the user to enter a security password code, 
one digit at a time, followed by the command OK. If the code is 
correct, the system sends an "arm" request to the security system, as 
shown in FIG. 3 j . When confirmation is received, the Entry Level 
screen is again displayed with the green "ready to arm" indicator 
replaced by a red "armed" indicator as shown in FIG. 3m. 

Detailed Description Paragraph Table - DETL (1): 

TABLE 1 Example of a 

Specific Task Description - Task 1: the Fire/Security Interface 

Data Validation: The data 

validation consists of the following states: 1st Synch byte: Must be 
equal to 52 or data is rejected 2nd Synch byte: Must be equal to 126 
or transmission is rejected Length byte: Receive length of message 
Command byte: Receive command byte Data bytes: Receive appropriate 
number of data bytes, if any Checksum byte: Must be equal to sum of 
all bytes in message MOD 256 or transmission is rejected Immediate 
Response: Call security device driver to send acknowledgment of 
receipt of message. Neutral State: If command is Fire Alarm, then set 
Master Touchscreen task to Fire Alarm State, jump to Fire Alarm State 
in Master Touchscreen Task, end if If command is Security Alarm, then 
set Master Touchscreen task to Security Intrusion State, jump to 
Security Intrusion State in Master Touchscreen Task end if If command 
is any other type of alarm, then call lighting moods device driver to 
turn on all lights end if ignore all other commands return to Polling 
Loop Contextual States: Waiting for response to disarm system 
request: if command is acknowledgment then result = good ]ump to 
Disarm S ec ur it y System substate of Master Touchscreen Task else if 
command is system already disarmed result = already disarmed jump to 
Disarm Securi ty System substate of Master Touchscreen Task else if 
command is password rejected result = password rejected jump to 
Disarm Security S y stem substate of Master Touchscreen Task else 
execute Neutral State without changing Contextual State end if 
Waiting for a response to shunt zone command: if command is 
acknowledgment then result = good jump to Shunt Zone substate of 
Master Touchscreen Task else if command is rejected password then 
result = rejected password jump to Shunt Zone substate of Master 
Touchscreen Task else execute Neutral State without changing 
Contextual State end if Waiting for interface to return zone map i f 
command is zone map then result = good jump to substate that 
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requested a zone map else if command is rejected password result = 
rejected password jump to substate that requested a zone map else 
execute Neutral State without changing Contextual State end if 
Waiting for response to arm system request: if command is 
acknowledgment then result = good jump to Arm Security System 
substate of Master Touchscreen Task else if command is system already 
armed result = already armed jump to Arm Security System substate of 
Master Touchscreen Task else if command is system not ready result = 
not ready jump to Arm Securit y S ystem substate of Master Touchscreen 
Task else if command is password rejected result = password rejected 
jump to Arm Security System substate of Master Touchscreen Task else 
execute Neutral State without changing Contextual State end if 



Detailed Description Paragraph Table - DETL (3): 

TABLE 3 Simple Device 

Drivers are as follows: Analog input driver Audio and video driver 
Bath and spa Bus digital input driver Digital output driver Lighting 
mood driver without acknowledgment Random lighting driver Vacation 
mode driver Voice recognition Voice synthesis (low priority) The 
following complex device drivers are available: Lighting mood driver 
with acknowledgment Security Interface Driver Serial Digital Input 
Driver Voice Synthesis (high priority) 



Detailed Description Paragraph Table - DETL (5): 

TABLE 5 Support 

Environment Functions: Display 

Cursor: Handles cursor movement and touchbox dynamics. Pull byte: 
Pulls a byte of data from a specified Serial Input Queue. 
Colorchange : Changes any color within a specified rec- tangular area 
to any other color, according to the values passed it. Inst imeevent : 
Inserts an event into the ISIS queue, according to the event type and 
event time passed it. Delt imeevent : Deletes an event from the ISIS 
queue, according to the event time passed it. Checktouch: Compares 
recognition templates and touch coordinates with the locations of the 
current touchboxes on the screen, and returns the number of any 
activated touchbox. Reloadsched: Reloads the daily schedule from a 
data file, screening out events that are not to occur on that day. 
Playafile: Speak a phrase. Playmemsequence : Speak a phrase followed 
by individual words, such as numbers, i.e. "The temp- erature is", 
"90", "5", "degrees". Loadscreen: Load a new menu screen and reset 
various graphics status variables. Loadslice: Load a portion of a 
screen. Loadvocab: Load a voice recognition vocabulary into a speech 
board. Initrecognit ion : Initialize voice recognition for a menu. 
Spchutil: A variety of routines designed to manage the speech boards 
in a cohesive, non-inter- fering manner. Commbyport : A general serial 
transmission routine. Avrsof terror : Error handler for code - detected 
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error conditions. Codeerror: Error handler for software run -time 
errors. Readana : Reads an analog voltage and/or temp- erature from an 
analog input. Readdigio: Reads the current- values of all the inter- 
rupting digital inputs. Metrabyteread : Reads the current values of a 
group of passive digital inputs according to the value passed it. 
Metrabyteon: Turns on a digital output according to the values passed 
it. Metrabyteof f : Turns off a digital output according to the values 
passed it. Metrapulseon : Pulses a digital output on for the duration 
passed it. Metrapulseof f : Pulses a digital output off for the 
duration passed it. 

Detailed Description Paragraph Table - DETL (6) : 

TABLE 6 *.pic: Full screen 

color graphics files in a format compatible with the hardware's color 
graphics controller board. These files are loaded by the HEART 
program into graphics memory to show a full-screen, high- resolut ion , 
color-graphics display on a monitor or touchscreen. * . f nt : Color 
graphics type fonts consisting of large characters and symbols to be 
dynamically displayed on the screen. *.icn: Icons and other graphics 
images designed to be displayed on the screen under program control. 
* . set : Multi-word or complete phrase Speech Response data files in 
the proper format for the speech response hardware. * . spk : Single 
word or partial phrase Speech Response data files in a format for 
appending with other .SPK files to build variable speech phrases 
compatible with the TISPEECH board. * . voc : Vocabulary template data 
compatible with the TISPEECH board's recognition mode. When these 
files are loaded into the TISPEECH board by the HEART program, they 
define the active recognition vocabulary. *.img: software files 
containing routines to be loaded for the TISPEECH board. Depending on 
which routine is loaded, the TISPEECH board will perform voice 
recognition, speech response, or telephone functions. 



Claims Text - CLTX (5) : 

audio communication interface means comprised of a voice 
recognition and speech system, through which the system and user 
communicate with each other, said voice recognition and speech system 
functioning together with said visual communication interface to 
provide cues to said user as to available commands, feedback of 
whether the current command has been accepted by the system and the 
results of performing said current command. 

Claims Text - CLTX (19) : 

a plurality of additional communication interface means, including 
at least two of a keypad, serial data keypad, a voice recognition 
system, hand-held remote control, computer keyboard and telephone, 
said at least two additional communication interface means being 
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simultaneously connected with each other and said visual 
communication interface means to said programmed data processor such 
that said user may communicate commands to said system using any o£ 
said connected communication interfaces. 

Claims Text - CLTX (26) : 

18. The system of claim 11, wherein the user may schedule the 
occurrence of user-determined events by the system by utilizing said 
voice reco g nition system. 



i 



^ a 1 :4343/C-/A?rS/EAST/cache/eas?0040804 1 5 1 629924 tmp 9 text_font<"ouricr% 8/4/0-1 



This Page is Inserted by IFW Indexing and Scanning 
Operations and is not part of the Official Record 



Defective images within this document are accurate representations of the original 
documents submitted by the applicant. 

Defects in the images include but are not limited to the items checked: 

□ BLACK BORDERS 

□ IMAGE CUT OFF AT TOP, BOTTOM OR SIDES 

□ fad!d TEXT OR DRAWING 



□ SKEWED/SLANTED IMAGES 

□ COLOR OR BLACK AND WHITE PHOTOGRAPHS 

□ GRAY SCALE DOCUMENTS 

□ LINES OR MARKS ON ORIGINAL DOCUMENT 

□ REFERENCE(S) OR EXHD3IT(S) SUBMITTED ARE POOR QUALITY 

□ OTHER: 

IMAGES ARE BEST AVAILABLE COPY. 
As rescanning these documents will not correct the image 
problems checked, please do not report these problems to 
the IFW Image Problem Mailbox. 



BEST AVAILABLE IMAGES 




BLURRED OR HXEGD3LE TEXT OR DRAWING 



